Focusing device and image pickup apparatus

ABSTRACT

A focusing device includes a distance information acquisition section acquiring information on a distance to a subject based on a plurality of image data acquired by taking images of the subject at a plurality of different viewpoint positions. A subject detection section detects a first area and a second area based on the distance information and threshold information for the distance range within which a subject of interest that obstructs a main subject in the subject is present. In the first area, the subject of interest is present in each of the plurality of image data. In the second area, the subject of interest is not present. A focusing estimation section estimates the focusing of the main subject based on the distance information on the second area in each of the plurality of image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2009-020164, filed Jan. 30, 2009,the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a focusing device and an image pickupapparatus configured to be able to focus on a subject even when anobstruction is present in front of the subject.

2. Description of the Related Art

An image pickup apparatus is used to take images of various scenes. Ifan obstruction is present between the image pickup apparatus and a mainsubject, the main subject may be hidden behind the obstruction,preventing the desired image from being obtained. For example, Jpn. Pat.Appln. KOKAI Publication No. 2001-43458 discloses a technique to removean obstruction by extracting a background image from the motion parallaxof the obstruction if the obstruction is, for example, a person or avehicle. Specifically, Jpn. Pat. Appln. KOKAI Publication No. 2001-43458discloses a technique to generate image data containing the backgroundfree from the obstruction by comparing a plurality of image dataobtained at different points in time with one another for each pixel,extracting pixels in which no change has been detected for apredetermined period of time, as background image data, and executingthe process of extracting background image data for, for example, allthe pixels in a monitor zone.

BRIEF SUMMARY OF THE INVENTION

A focusing device according to a first aspect of the present inventioncomprises a distance information acquisition section configured toacquire information on a distance to a subject based on a plurality ofimage data acquired by taking images of the subject at a plurality ofdifferent viewpoint positions; a subject detection section configured todetect a first area and a second area based on the distance informationobtained by the distance information acquisition section and thresholdinformation for the distance range within which a subject of interestthat obstructs a main subject in the subject is present, the subject ofinterest being present in the first area in each of the plurality ofimage data, the subject of interest not being present in the secondarea; and a focusing estimation section configured to carry out thefocusing estimation on the main subject based on the distanceinformation on the second area in each of the plurality of image data.

An image pickup apparatus according to a second aspect of the presentinvention comprises, a distance information acquisition sectionconfigured to acquire information on a distance to a subject based on aplurality of image data acquired by taking images of the subject at aplurality of different viewpoint positions; a subject detection sectionconfigured to detect a first area and a second area based on thedistance information obtained by the distance information acquisitionsection and threshold information for the distance range within which asubject of interest that obstructs a main subject in the subject ispresent, the subject of interest being present in the first area in eachof the plurality of image data, the subject of interest not beingpresent in the second area; a focusing estimation section configured tocarry out the focusing estimation on the main subject based on thedistance information on the second area in each of the plurality ofimage data; and an image generating section configured to acquire imagedata by taking images of the subject including the main subject on whichthe focusing estimation has been carried out by the focusing estimationsection.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a diagram showing the configuration of a first embodiment ofan image pickup apparatus that is a twin-lens stereo camera comprising afocusing device according to the present invention;

FIG. 2 is a diagram showing the configuration of an image input sectionprovided in the image pickup apparatus instead of two image pickupsections;

FIG. 3 is a flowchart showing a basic operation performed by a distanceinformation acquisition section of the image pickup apparatus tocalculate distance information;

FIG. 4 is a flowchart showing the basic operation of a subject detectionsection of the image pickup apparatus;

FIG. 5 is a schematic diagram showing a first area in which a subject ofinterest that obstructs a main subject is present and a second area inwhich the subject of interest is not present, in each of a plurality ofimage data consecutively acquired by the image pickup apparatus as timeelapses;

FIG. 6A is a schematic diagram showing one of the images in two framesacquired first by an image input section in a second embodiment of animage pickup apparatus comprising a focusing device according to thepresent invention;

FIG. 6B is a schematic diagram showing the other of the images in thetwo frames acquired first by the image pickup apparatus;

FIG. 7 is a flowchart of the basic operation of an image generatingsection of the image pickup apparatus;

FIG. 8A is a schematic diagram showing image data on a person presentbehind an obstruction separated by a third embodiment of an image pickupapparatus comprising a focusing device according to the presentinvention;

FIG. 8B is a schematic diagram showing image data on the obstructionseparated by the image pickup apparatus and on the forward side of theobstruction; and

FIG. 9 is a schematic diagram showing image data generated by imagegenerating means of the image pickup apparatus and which is free fromthe obstruction.

DETAILED DESCRIPTION OF THE INVENTION

A first embodiment of the present invention will be described below withreference to the drawings.

FIG. 1 is a diagram showing the configuration of an image pickupapparatus of a twin-lens stereo camera including a focusing device. Thesubject includes both a main subject and a background subject (subjectof interest, the subject other than the main subject). The main subjectis, for example, a person 11 as shown in FIG. 6A. The subject ofinterest is an obstruction such as a fence 12.

The present apparatus 10 includes, for example, two image pickupsections 100 and 200 as an image input section. The present apparatusincludes a distance information acquisition section 300, a subjectdetection section 400, an image generating section 500, a temporarystorage section 600, a display section 700, a storage section 800, and asetting section 900.

The two image pickup sections 100 and 200 pick up images at respectivedifferent viewpoint positions to acquire image signals for a pluralityof frames for each viewpoint position. The image pickup sections 100 and200 form a stereo camera. The image pickup section 100 includes anoptical system 101, an image pickup element 102, and storage section103. The image pickup section 200 includes an optical system 201, animage pickup element 202, and storage section 203.

Each of the optical systems 101 and 201 focuses a light flux from thesubject to form an image on the corresponding one of the image pickupelements 102 and 202. Each of the image pickup elements 102 and 202converts the subject image formed by the corresponding one of theoptical systems 101 and 102, into an analog electric signal and thenconverts the analog electric signal into a digital signal (pixelsignal). The digital signal (image signal) is stored in thecorresponding one of the storage section 103 and 203. Each of thestorage section 103 and 203 is configured to temporarily store the imagesignal provided by the corresponding one of the image pickup elements102 and 202.

The apparatus 10 is configured as a twin-lens stereo camera includingthe two image pickup sections 100 and 200. The present apparatus 10 isnot limited to the two image pickup sections 100 and 200 but mayincludes, for example, at least three image pickup sections. The presentapparatus 10 may includes at least one image pickup section configuredto take an image a number of times with a viewpoint position changed.

The present apparatus 10 is also applicable to, for example, an imageprocessing apparatus in which such an image processing program as shownin FIG. 2 is installed. The image processing apparatus 20 basically hassubstantially the same configuration as that of the image pickupapparatus 10 shown in FIG. 1. The image processing apparatus 20 isdifferent from the image pickup apparatus 10 in that an image inputsection 100 a is provided instead of, for example, the two image pickupsections 100 and 200.

The image input section 100 a acquires image signals for a plurality offrames acquired by image taking at respective different viewpointpositions. The image input section 100 a includes any storage medium inwhich image signals for a plurality of frames are pre-stored. The imageinput section 100 a may also provide the function of outputting imagesignals for a plurality of frames to the storage medium. The storagesection 800 may have the function of the image input section 100 a.

The distance information acquisition section 300 uses image signals forN frames (N≧2) provided by each of the two image pickup sections 100 and200 to acquire information on the distance to the subject in pixel unit.The distance information acquisition section 300 acquires information onthe distance to the subject based on a plurality of image data acquiredat a plurality of different viewpoint positions, for example, aplurality of image data acquired by each of the two image pickupsections 100 and 200.

The distance information acquisition section 300 sequentially acquiresdistance information for each of a plurality of image consecutivelyacquired at a plurality of different viewpoint positions, for example,each of a plurality of image data obtained by each of the two imagepickup sections 100 and 200. A set of distance information acquired bythe distance information acquisition section 300 in pixel unit ishereinafter referred to as distance image data.

The distance information acquisition section 300 references areainformation indicating whether or not the subject of interest extractedby the subject detection section 400 is present to execute a focusingestimation process again.

The subject detection section 400 uses distance image data acquired bythe distance information acquisition section 300 to detect an area inwhich the subject of interest is present. Specifically, the subjectdetection section 400 receives distance threshold information set by aphotographer to binarize information on the distance to the subjectacquired by the distance information acquisition section 300, based onthe threshold information. The subject detection section 400 thusacquires binary information for each pixel.

The subject detection section 400 detects a first area and a second areabased on the distance information and threshold information acquired bythe distance information acquisition section 300. The thresholdinformation indicates the distance range within which the subject ofinterest obstructing the main subject is present. In the first area, thesubject of interest is present in each of the plurality of image data.In the second area, the subject of interest is not present.

The subject detection section 400 detects the first area and the secondarea in each of the plurality of consecutively acquired image data.

The distance information acquisition section 300 estimates the focusingof the main subject based on the distance information on the second areain each of the plurality of image data obtained by, for example, the twoimage pickup sections 100 and 200 in which area the subject of interestis not present (focusing estimation section). The distance informationacquisition section 300 sequentially estimates the focusing of thefocusing of the main subject based on the distance information on thesecond area sequentially detected by the subject detection section 400(focusing estimation section). The distance information acquisitionsection 300 uses image data on the second area in which the subject ofinterest is not present to estimate the focusing of the main subject bymeans of hill climbing (focusing estimation section).

The image generating section 500 executes predetermined image processingbased on area information extracted by the subject detection section 400is present and indicating whether or not the subject of interest.

The temporary storage section 600 temporarily stores data processed bythe distance information acquisition section 300, the subject detectionsection 400, and the image generating section 500.

The display section 700 displays various image data.

The storage section 800 stores image data acquired by the imagegenerating section 500.

The setting section 900 is an operation section operated by thephotographer to make various settings, for example, to set an operationmode for face detection in which the face site of a human bodycorresponding to the subject, is detected.

The specific operation of the distance information acquisition section300 will be described with reference to a flowchart of a basic operationfor roughly calculating distance information as shown in FIG. 3. The twoimage pickup sections 100 and 200 are hereinafter referred to as imageinput means 100 and 200.

Image data on a plurality of frames from the image input sections 100and 200 is input to the distance information acquisition section 300.Then, in step S301, the distance information acquisition section 300sets an area for which distance information is acquired from the imagedata on the plurality of frames. The distance information acquisitionarea may be set, for example, by the photographer by operating thesetting section 900 or automatically by the distance informationacquisition section 300.

In step S302, for the set distance information acquisition area, thedistance information acquisition section 300 calculates correspondingpoints among the image data for the plurality of frames by an imagecorrelation method. The distance information acquisition section 300stores a correlation parameter for the calculated corresponding pointsin the temporary storage section 600.

In step S303, the distance information acquisition section 300calculates information on the distance to the subject for each pixelbased on the correlation parameter for the corresponding points.

In step S304, the distance information acquisition section 300 acquiresdistance image data based on the calculated information on the distanceto the subject for each pixel. The distance information acquisitionsection 300 stores the distance image data in the temporary storagesection 600. The distance information acquisition section 300 acquiresinformation on the distance to the subject based on the plurality ofimage data obtained at different viewpoint positions, for example, theplurality of image data obtained by each of the two image pickupsections 100 and 200. The distance information acquisition section 300sequentially acquires distance information for each of the plurality ofimage data consecutively acquired at different viewpoint positions, forexample, the plurality of image data obtained by each of the two imagepickup sections 100 and 200.

The specific operation of the subject detection section 400 will bedescribed with reference to a flowchart of the basic operation of thesubject detection section 400 shown in FIG. 4.

In step S401, the subject detection section 400 receives distancethreshold information set by the photographer from, for example, thesetting section 900. The threshold information is set, for example, bythe photographer by operating the setting section 900.

In step S402, upon receiving the set threshold information, the subjectdetection section 400 reads distance image data stored in the temporarystorage section 600. The subject detection section 400 compares themagnitude of the distance information for each pixel in the distanceimage data with the magnitude of the threshold information. Inaccordance with the magnitudes of the distance information and thethreshold information, the distance information for each image in thedistance image data is binarized.

In step S403, the subject detection section 400 binarizes the distanceinformation for each pixel based on the threshold information and thenstores the binary information for each pixel in the temporary storagesection 600. The subject detection section 400 detects a first area anda second area based on the distance information and thresholdinformation acquired by the distance information acquisition section300. The threshold information indicates the distance range within whichthe subject of interest that obstructs the main subject is assumed to bepresent. In the first area in each of the plurality of image data, thesubject of interest is present. In the second area, the subject ofinterest is not present. The subject detection section 400 sequentiallydetects the first area and the second area for every pluralconsecutively-acquired image data.

FIG. 5 schematically shows the first area and second area for everyplural image data consecutively acquired as time elapses from t1 to tn,for example, for every plural image data obtained by each of the twoimage pickup sections 100 and 200. A plurality of image data L1 to Lnobtained by the image pickup section 100 contain respective first areasgal to gan in which the subject of interest is present and respectivesecond areas fa1 to fan in which the subject of interest is not present.A plurality of image data R1 to Rn obtained by the image pickup section200 contain respective first areas gb1 to gbn in which the subject ofinterest is present and respective second areas fb1 to fbn in which thesubject of interest is not present.

The distance information acquisition section (focusing estimationsection) 300 detects focusing in an image signal for a non-obstructingarea portion in the image data for the plurality of frames based on thebinary information for each pixel stored in the temporary storagesection 600. The focusing detection processing is based on the mostcommon technique called hill climbing or a contrast method and widelyused for electronic image pickup apparatuses such as digital cameras.

The distance information acquisition section (focusing estimationsection) 300 estimates the focusing of the main subject based on thedistance information on the second areas fa1 to fan and fb1 to fbn inthe plurality of image data obtained by, for example, the two imagepickup sections 100 and 200 as shown in FIG. 5.

The distance information acquisition section (focusing estimationsection) 300 sequentially estimates the focusing of the main subjectbased on the distance information on the second areas fa1 to fan and fb1to fbn sequentially detected by the subject detection section 400.

The distance information acquisition section (focusing estimationsection) 300 uses the image data on the second areas fa1 to fan and fb1to fbn in which the subject of interest is not present to estimate thefocusing of the main subject by hill climbing.

For example, the distance information acquisition section (focusingestimation section) 300 first estimates the focusing of the main subjectbased on the distance information on the second areas fa1 and fb1sequentially detected by the subject detection section 400 at points intime t1 to to as time elapses as shown in FIG. 5. FIG. 5 schematicallyshows that the focusing of the main subject is estimated based on thedistance information in image data M1 composed of the second areas fa1and fb1 in which the subject of interest is not present at point in timet1.

If no focusing of the main subject is detected at point in time t1, thenat point in time t2, the distance information acquisition section(focusing estimation section) 300 estimates the focusing of the mainsubject based on the distance information on the second areas fa2 andfb2 sequentially detected by the subject detection section 400. At thistime, the distance information acquisition section (focusing estimationsection) 300 estimates the focusing of the main subject based on thedistance information in image data M2 composed of the second areas fa2and fb2 in which the subject of interest is not present.

Similarly, until the focusing of the main subject is detected, thedistance information acquisition section (focusing estimation section)300 estimates, between time t3-tn, the focusing of the main subjectbased on the distance information on the second areas fa2 to fan and fb2to fbn sequentially detected by the subject detection section 400.

The basic configuration and operation of a focusing detection processbased on hill climbing will be described.

Each of the optical systems 101 and 201 includes a plurality of lensgroups. The plurality of lens groups are intended to take images, forexample, groups of taking lenses. Some of the plurality of lens groupscan be driven in the direction of the optical axis in order to adjustthe focus. The lenses of these lens groups are called focus lenses.

Images of the subject formed by the optical systems 101 and 201 areconverted into electric signals by photoelectric conversion elements ofthe image pickup elements 102 and 202.

The focus lenses of the optical systems 101 and 201 are moved in thedirection of the optical axis. An evaluation value for blur in aplurality of image data taken during the movement of the focus lenses iscalculated. The contrast in the image data and the sum of high frequencycomponents is used as the evaluation value. A larger evaluation valueindicates that the subject is in better focus.

The distance information acquisition section 300 estimates focusing bypeak estimation. The distance information acquisition section (focusingestimation section) 300 estimates focusing utilizing the fact that alarger evaluation value indicates that the subject is in better focus.

Thus, according to the present invention, information on the distance tothe subject is obtained based on a plurality of image data taken at aplurality of different viewpoint positions. Based on the distanceinformation and the threshold information on the distance range withinwhich the subject of interest that obstructs the main subject is assumedto be present, the first area and the second area are detected. Then,based on the distance information on the second area in which thesubject of interest is not present, the focusing of the main subject isestimated. Thus, even if the obstruction is stationary, the subject canbe appropriately brought into focus.

Now, a second embodiment of the present invention will be described. Animage pickup apparatus according to the present embodiment has the sameconfiguration as that of the image pickup apparatus according to theabove-described first embodiment. Thus, the second embodiment will alsobe described with reference to FIG. 1, described above.

In the present apparatus, a subject detection section 400 acquiresbinary information, and the subject is then detected by imagerecognition, for example, a face detection process is carried out inwhich the face site of a human body corresponding to the subject isdetected.

The subject detection section 400 generates a plurality of evaluationvalues using, as indices, the similarities of a plurality of featurepoints of the main subject in a plurality of the image data. The subjectdetection section 400 then integrates the plurality of evaluation valuestogether to determine whether or not the second area corresponds to themain subject based on the integrated evaluation values for the secondarea (image recognition section).

The subject detection section (image recognition section) 400 generatesand integrates evaluation values for the appropriate areas of the mainsubject, for example, the face site of a human body corresponding to thesubject. Before generating and integrating the evaluation values for theappropriate areas of the main subject, for example, the face site of thehuman body, the subject detection section (image recognition section)400 determines the evaluation values for the obstructing area with thesubject of interest to indicate a part of the main subject.

The setting section 900 is set to a face detection operation mode. Thepresent apparatus 10 includes a database in which the feature points ofthe face site are registered. The database may be formed in, forexample, the storage section 800.

The subject detection section (image recognition section) 400 detects,for example, the face site of a person 11 corresponding to the subject,by means of image recognition. That is, when the face detectionoperation mode is set, the subject detection section (image recognitionsection) 400 normalizes the image data and the feature points based onimage data acquired by the image input sections 100 and 200 and dataindicative of the feature points of the face site pre-registered in thedatabase. Thereafter, the subject detection section (image recognitionsection) 400 uses a predetermined filter process corresponding to apattern collation process to generate evaluation values using, as anindex, the similarity between the image data and each of the featurepoints. The evaluation values do not allow the face site correspondingto the main subject at the time of generation of the evaluation valuesto be recognized but are the results of collation for the face site orin smaller unit sites. Thus, a plurality of evaluation values aregenerated for one face site.

After acquiring the binary information, the subject detection section(image recognition section) 400 integrates the evaluation valuestogether based on, for example, the components of the face site of theperson 11, such as the eyes, the nose, and the face line. Based on thereliability (similarity) of the area resulting from the integrationprocess, the subject detection section (image recognition section) 400determines whether or not the area corresponds to the face site. Upondetermining that the area corresponds to the face site, the subjectdetection section (image recognition section) 400 stores coordinateinformation on the area determined to correspond to the face site, inthe temporary storage section 600.

When a common face detection process is carried out with the mainsubject (face site) obstructed by, for example, an obstacle that is anobstruction, the reliability (similarity) of the area resulting from theintegration process may not be accurately owing to the adverse effect ofthe obstructing area.

After acquiring the binary information and before integrating theevaluation values together based on, for example, the components of theface site such as the eyes, the nose, and the face line, the subjectdetection section 400 carries out a uniquely determining process, forexample, determines all the evaluation values for the obstructing areato indicate a part of the face site, based on the binary information foreach pixel. After the determination, the above-described integrationprocess is carried out.

The operation of the image pickup apparatus will be described.

Images acquired first by the image input sections 100 and 200 arespecifically assumed to be image data for two frames shown in FIGS. 6Aand 6B. The image data shown in FIG. 6A serves as a criterion when theimage correlation method is carried out to calculate a distance image(this image data is hereinafter referred to as criterion image data Ds).The image data shown in FIG. 6B is referenced when the image correlationmethod is carried out to calculate a distance image (this image data ishereinafter referred to as reference image data Dr).

In both the criterion image data Ds and the reference image data Dr, aperson 11 is a main subject, and a fence 12 or the like present in frontof the person 11 is an obstruction against the main subject.

The present apparatus 10 calculates distance image data using thecriterion image data Ds and the reference image data Dr. Based on thecalculated distance image data, the present apparatus 10 detects thesubjects using the obstruction such as the fence 12 as a subject ofinterest. The present apparatus 10 thus separates the criterion imagedata Ds into an area in which the obstruction such as the fence 12corresponding to the subject of interest, is present and an area inwhich the obstruction is not present. Thereafter, the present apparatus10 estimates the focusing of the area in which the obstruction such asthe fence 12 is not present.

Now, the operation will be specifically described.

The distance information acquisition section 300 acquires distance imagedata in accordance with the flowchart of the basic operation for roughlycalculating distance information as shown in FIG. 3. Before acquiringdistance image data, the photographer operates the setting section 900to select the criteria image data Ds and the reference image data Dr andsets a distance image acquisition area in the criteria image data Ds.For example, the distance information acquisition area is set to be theentire area of the criteria image data Ds shown in FIG. 6A.

In response to the setting operation performed on the setting section900, in step S302, the distance information acquisition section 300 usesthe image correlation method to calculate, for the set distance imageacquisition area, the corresponding points between the image data forthe plurality of frames. The image correlation method involves, forexample, calculating the amount of the correlation between the imagedata. The distance information acquisition section 300 stores thecalculated correlation parameters for the corresponding points, in thetemporary storage section 600.

In step S303, based on the correlation parameters for the correspondingpoints, the distance information acquisition section 300 calculatesinformation on the distance to the subject for each pixel in thedistance information acquisition area in the criterion image data Ds.

In step S304, the distance information acquisition section 300 storesthe distance information in the temporary storage section 600 asdistance image data including the subject distance. The distanceinformation acquisition section 300 stores information on thecorrespondence between the pixel position (XY coordinates) and subjectdistance in the distance information acquisition area in the criterionimage data Ds in the temporary storage section 600.

That is, as described above, the distance information acquisitionsection 300 acquired information on the distance to the subject based ona plurality of image data taken at a plurality of different viewpointpositions, for example, a plurality of image data obtained by each ofthe two image pickup sections 100 and 200.

The distance information acquisition section 300 sequentially acquiresdistance information for each of a plurality of image data taken at aplurality of consecutively-acquired different viewpoint positions, forexample, a plurality of image data obtained by each of the two imagepickup sections 100 and 200. The distance information acquisitionsection 300 then stores the distance information in the temporarystorage section 600 as distance image data including the subjectdistance.

The subject detection section 400 detects the subject of interest inaccordance with a flowchart in FIG. 4 showing the basic operation of thesubject detection section 400.

In step S401, the subject detection section 400 receives distancethreshold information set by the photographer from, for example, thesetting section 900. The threshold information is set, for example, bythe photographer by operating the setting section 900. Specifically, thephotographer sets the distance range within which the subject ofinterest corresponding to the fence 12 or the like, is assumed to bepresent, that is, the same distance as that to the fence 12 or adistance slightly longer than that to the fence 12, to be thresholdinformation. The photographer changes focal distance, for example,changes scale such that the vicinity of the obstruction, for example,the fence 12 is focused on.

Upon receiving the operation from the setting section 900, the subjectdetection section 400 automatically calculates the distance to thesubject from the margin of the threshold and the focal distances of theoptical systems 101 and 201. The subject detection section 400 then setsthe distance to the subject to be distance threshold information. Themethod for determining the focal distance is not limited to opticalzooming but any method for changing the focal distance, such as a zoomscale or AF control, may be used. Setting a threshold for each imagetaking operation may be avoided by storing threshold information setduring the preceding image taking operation and using this thresholdinformation when the scale is not changed.

In step S402, upon receiving the set threshold information, the subjectdetection section 400 reads the distance image data stored in thetemporary storage section 600. The subject detection section 400compares the magnitude of the distance information for each pixel in thedistance image data with the magnitude of the threshold information. Thesubject detection section 400 thus determines whether or not the subjectdistance for each pixel in the distance information acquisition area iswithin a predetermined range. The subject detection section 400 executesa binarization process in accordance with the magnitudes of the distanceinformation and the threshold information.

In step S403, the subject detection section 400 binarizes the distanceinformation for each pixel based on the threshold information and thenstores the binary information for each pixel, that is, the binaryinformation indicating whether or not the subject of interest(obstruction) is present, in the temporary storage section 600.

The threshold information is set to, for example, the same distance asthat from the position of the photographer to the fence 12 or a distanceslightly longer than that from the position of the photographer to theposition of the fence 12. If the subject distance is less than or equalto the threshold, this means that the fence 12 corresponding to thesubject of interest, is present in front of the person 11 correspondingto the main subject.

The subject detection process executed by the present apparatus 10 isrepeated every time the image data from each of the image input sections100 and 200 is updated. Binary information is thus accumulated in thetemporary storage section 600. As described above, based on the distanceinformation acquired by the distance information acquisition section 300and the threshold information on the distance range within which thesubject of interest that obstructs the main subject is assumed to bepresent, the subject detection section 400 detects the first area inwhich the subject of interest is present and the second area in whichthe subject of interest is not present, for every plural image dataconsecutively acquired as time elapses.

The subject detection section 400 carries out the face detection processof detecting, for example, the face site of the person 11 correspondingto the subject. The setting section 900 is, for example, operated by thephotographer so as to set the face detection operation mode.

When the face detection operation mode is set, the subject detectionsection 400, then based on the criterion image data Ds and referenceimage data Dr acquired by the image input means 100 and 200 through animage pickup operation and data indicative of the feature points of theface site pre-registered in the database (storage section 800), thesubject detection section 400 normalizes the criterion image data Ds andreference image data Dr and the feature points.

The subject detection section 400 carries out the predetermined filterprocess on each of the feature points and the criterion image data Dsand reference image data Dr, the filter process corresponding to thepattern collation process to generate evaluation values using, as anindex, the similarity between the feature point and the criterion imagedata Ds and reference image data Dr.

The subject detection section 400 integrates the evaluation valuestogether based on, for example, the components of the face site, such asthe eyes, the nose, and the face line. Based on the reliability(similarity) of the area resulting from the integration process, thesubject detection section 400 determines whether or not the areacorresponds to the face site. Upon determining that the area correspondsto the face site, the subject detection section 400 stores coordinateinformation on the area determined to correspond to the face site, inthe temporary storage section 600.

At this time, before integrating the evaluation values together basedon, for example, the components of the face site such as the eyes, thenose, and the face line, the subject detection section 400 carries out auniquely determining process, for example, determines all the evaluationvalues for the obstructing area to indicate a part of the face site,based on the binary information for each pixel. After the determination,the subject detection section 400 carries out the above-describedintegration process. This allows the adverse effect of the obstructingarea to be eliminated when the apparatus determines whether or not thearea corresponds to, for example, the face site based on the reliability(similarity) of the area after the integration process.

The distance information acquisition section 300 carries out a focusingestimation process on the face site recognized by the subject detectionsection 400. That is, images of the subject formed by the opticalsystems 101 and 201 are converted into analog electric signals by thephotoelectric conversion elements of the respective image pickupelements 102 and 202. The analog electric signals are then convertedinto digital signals (pixel signals). The digital signals (imagesignals) are stored in the storage section 103 and 203 as image data.

At this time, the focus lenses of the optical systems 101 and 201 aremoved in the direction of the optical axis. During the movement of thefocus lenses, a plurality of image data are acquired again. In thepresent embodiment, in a strict sense, the plurality of image dataacquired with the focus lenses in motion are taken at different pointsin time and different angles of view. However, for convenience ofdescription, it is assumed that the amount of blue varies among theimage data as shown in the criterion image data Ds and reference imagedata Dr in FIGS. 6A and 6B.

Based on, for example, the criterion image data Ds and reference imagedata Dr reacquired and the binary information corresponding to thecriterion image data Ds and the reference image data Dr, the distanceinformation acquisition section 300 selects, from the criterion imagedata Ds, an image signal for the face site in which no obstruction suchas the fence 12 is present and which has been detected by theabove-described face detection process. The distance informationacquisition section 300 uses the selected image signal for the face siteto estimate focusing by means of hill climbing.

The distance information acquisition section (focusing estimationsection) 300, for example, estimates the focusing of the main subjectbased on the distance information on the second areas fa1 to fan and fb1to fbn in which the subject of interest is not present, in the pluralityof image data obtained by each of the two image pickup sections 100 and200, as shown in FIG. 5. The distance information acquisition section(focusing estimation section) 300 sequentially estimates the focusing ofthe main subject based on the distance information on the second areasfa1 to fan and fb1 to fbn sequentially detected by the subject detectionsection 400. In this case, the distance information acquisition section(focusing estimation section) 300 uses the image data on the secondareas fa1 to fan and fb1 to fbn in which the subject of interest is notpresent to estimate the focusing of the main subject by hill climbing.

Thus, according to the above-described second embodiment, the subjectdetection section 400 estimates the focusing of the areas in which noobstruction such as the fence 12 is present, as follows. Based on thecriterion image data Ds and reference image data Dr acquired by the twoimage pickup sections 100 and 200 and the data indicative of the featurepoints of the face site, the subject detection section 400 recognizesthe face site of the person 11 corresponding to the main subject, forexample, the eyes, the nose, or the face line. The subject detectionsection 400 then estimates the focusing of the recognized face site.

Since the focusing estimation is performed after the face site of theperson 11 corresponding to the main subject, for example, the eyes, thenose, or the face line, has been recognized, the focusing of the person11 corresponding to the main subject can be accurately estimated.Furthermore, the focusing estimation can be achieved without beingaffected by the obstruction such as the fence 12 which obstructs imagetaking of the person 11. Thus, even if a part of the main subject isobstructed by the obstruction such as the fence 12, an image can beacquired with the main subject accurately in focus.

Now, a third embodiment of the present invention will be described. Animage pickup apparatus according to the present embodiment has the sameconfiguration as that of the image pickup apparatus according to theabove-described first embodiment. Thus, the third embodiment will alsobe described with reference to FIG. 1, described above.

The image generating section 500 allows the distance informationacquisition section (focus estimation section) 300 to estimate focusing.The image generating section 500 then separates one of a plurality ofimage data into two image data based on the position of the subject ofinterest. The image generating section 500 carries out image processingon the two image data to acquire image data free from the subject ofinterest.

In the present embodiment, for example, the criterion image data Ds andreference image data Dr shown in FIGS. 6A and 6B are used to obtain thedistance image data. Then, based on the distance image data, the subjectsuch as the person 11 is automatically detected with the obstructionsuch as the fence 12 set to be a subject of interest. Thus, thereference image data is separated into the area in which the obstruction12 corresponding to the subject of interest is present and the area inwhich the obstruction 12 is not present. Thereafter, first imageprocessing α is carried out on the area in which the obstruction 12corresponding to the subject of interest is present. Second imageprocessing β that is different from the first image processing α iscarried out on the area in which the obstruction 12 is not present.

Specifically, the image generating section 500 carries out thepredetermined image processing based on the area information indicatingwhether or not the subject of interest extracted by the subjectdetection section 400 is present. The image generating section 500separates the image data obtained by each of the image pickup sections100 and 200 into two areas based on binary information acquired by thesubject detection section 400. The image generating section 500 carriesout the first image processing α on the area in which the subject ofinterest such as the fence 12 is present. The image generating section500 carries out the second image processing β, which is different fromthe first image processing α, on the area in which the subject ofinterest such as the fence 12 is not present. The image generatingsection 500 then integrates the image data subjected to the first imageprocessing α with the image data subjected to the second imageprocessing β, to acquire one image data.

The image generating section 500 operates in accordance with a flowchartof a basic operation shown in FIG. 7. The temporary storage section 600stores the binary information acquired by the subject detection section400 and the criterion image data Ds and reference image data Dr acquiredby the image input section 100 a and shown in, for example, FIGS. 6A and6B.

In step S501, the image generating section 500 reads the binaryinformation and criterion image data Ds stored in the temporary storagesection 600. The image generating section 500 separates the readcriterion image data Ds into two areas based on the binary information.

In step S502, the image generating section 500 carries out the firstprocessing α on the area resulting from the separation and in which thesubject of interest is present. In step S503, the image generatingsection 500 carries out the second processing β, which is different fromthe first processing α, on the area in which the subject of interest isnot present.

In step S504, the image generating section 500 integrates the image datasubjected to the first image processing α with the image data subjectedto the second image processing β, to acquire one image data.

In step S505, the image generating section 500 displays the image dataresulting from the integration, on the display section 700. The imagegenerating section 500 carries out output processing such as storage ofthe image data resulting from the integration in the storage means 800.

Now, description will be given of an operation of removing theobstruction such as the fence 12 after focusing estimation has beenperformed by the distance information acquisition section 300 accordingto the above-described first embodiment.

After focusing estimation is performed by the distance informationacquisition section (focusing estimation section) 300, the imagegenerating section 500 separates, for example, the criterion image dataDs shown in FIG. 6A into two areas based on the binary image dataacquired by the subject detection section 400. The binary informationindicates whether or not the obstruction such as the fence 12 is presentin front of the person 11 corresponding to the main subject. Thus, ifthe criterion image data Ds is separated based on the binaryinformation, the criterion image data Ds is separated into, for example,image data Ds-1 showing the person 11 or the like present behind theobstruction such as the fence 12 as shown in FIG. 8A and image data Ds-2showing the obstruction such as the fence 12 and the forward side of thefence 12 as shown in FIG. 8B.

After the criterion image data Ds is separated into the two image dataDs-1 and Ds-2, the image generating section 500 carries out the firstimage processing α on the obstructing area in which the obstruction suchas the fence 12 corresponding to the subject of interest is present asshown in FIG. 8A. In the first image processing α, an image signal(first image signal) for the obstruction such as the fence 12 in theobstructing area is determined to be a defect pixel. The image signalfor the defective pixel is obtained by interpolating image signals forsurroundings in the same frame.

The second image processing β does not particularly involve processingrelated to the removal of the obstruction such as the fence 12.

The image generating section 500 integrates the image data Ds-1 shown inFIG. 8A with the image data Ds-2 shown in FIG. 8B and subjected to thereplacement process, to acquire, for example, image data Dt such as theone shown in FIG. 9. The image data Dt does not contain the obstructionsuch as the fence 12 which otherwise obstructs image taking of theperson 11. Furthermore, the image data Dt is generated by interpolatingthe image data Ds-1 with another image data for the removed obstructionsuch as the fence 12, for example, the image data Ds-2.

The image generating section 500 outputs the image data Dt by, forexample, allowing the display section 700 to display the image data Dtor storing the image data Dt in the storage means 800. In the firstimage processing α, the obstruction such as the fence 12 may be removedby replacement with a particular color described in, for example, Jpn.Pat. Appln. KOKAI Publication No. 2006-30192 and interpolation that usesat least three images.

Thus, according to the third embodiment, the focusing of the area inwhich the obstruction such as the fence 12 is not present is estimatedas follows. To estimate the focusing of the main subject based on thedistance information on the second areas fa1 to fan and fb1 to fbn inwhich the subject of interest is not present, the subject detectionsection 400 recognizes the face site of person 11 corresponding to themain subject, for example, the eyes, the nose, or the face line, basedon the criterion image data Ds and reference image data Dr acquired bythe two image pickup sections 100 and 200 and the data indicative of thefeature points of the face site. The subject detection section 400estimates the focusing of the recognized face site and then performs anoperation of removing the obstruction.

As described above, as is the case with the above-described firstembodiment, the focusing estimation is performed after the face site ofperson 11 corresponding to the main subject, for example, the eyes, thenose, or the face line, has been recognized. Thus, the focusing of theperson 11 corresponding to the main subject can be accurately estimated.Then, such image data Dt as shown in FIG. 8 can be generated by removingthe obstruction such as the fence 12 and interpolating the image datawith another image signal for the removed image portion.

Thus, the predetermined settings are finished before image taking so asto simplify operations required for the image taking. This enables theimage taking to be achieved by conventional operations. Consequently,even with the presence of the obstruction such as the fence 12, imagescan be taken with the main subject such as the person 11 appropriatelyin focus. As a result, the image data Dt free from the obstruction suchas the fence 12 can be provided.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

For example, in the above-described embodiments, the image pickupapparatus in the twin-lens stereo camera comprises the image processingdevice. However, the present invention is not limited to this aspect.The present invention may be applied to an image processing device towhich a plurality of image data taken at a plurality of differentviewpoint positions and which, based on the input image data, acquiresimage data free from the obstruction such as the fence 12.

Furthermore, the subject is not limited to the person 11 but may be, forexample, a vehicle such as a traveling train or car, or a flying object.

The subject detection section 400 recognizes the face site based on thedata indicative of the feature points of the pre-registered face site,for example, the components of the face site such as the eyes, the nose,and the face line. However, the present invention is not limited to thisaspect. The recognition may be carried out based on a characteristicportion, for example, a uniform number.

1. A focusing device comprising: a distance information acquisitionsection configured to acquire information on a distance to a subjectbased on a plurality of image data acquired by taking images of thesubject at a plurality of different viewpoint positions; a subjectdetection section configured to detect a first area and a second areabased on the distance information acquired by the distance informationacquisition section and threshold information for the distance rangewithin which a subject of interest that obstructs a main subject in thesubject is present, the subject of interest being present in the firstarea in each of the plurality of image data, the subject of interest notbeing present in the second area; and a focusing estimation sectionconfigured to carry out the focusing estimation on the main subjectbased on the distance information on the second area in each of theplurality of image data.
 2. The focusing device according to claim 1,wherein the distance information acquisition section sequentiallyacquires the distance information for each of the plurality of imagedata acquired by consecutively taking images of the subject at theviewpoint positions, the subject detection section detects the firstarea and the second area in each of the plurality of consecutivelyacquired image data, and the focusing estimation section sequentiallycarries out the focusing estimation on the main subject based on thedistance information on the second area sequentially detected by thesubject detection section.
 3. The focusing device according to claim 1,wherein the focusing estimation section carries out the focusingestimation on the main subject by hill climbing using the image data onthe second area.
 4. The focusing device according to claim 1, furthercomprising: an image recognition section configured to generate aplurality of evaluation values using, as an index, similarity of each ofa plurality of feature points of the main subject in the plurality ofimage data, integrate the evaluation values together, and recognizewhether or not the second area corresponds to the main subject based onthe evaluation values for the second area after the integration process.5. The focusing device according to claim 1, wherein the imagerecognition section generates and integrates the evaluation values foreach site of the main subject together.
 6. The focusing device accordingto claim 4, wherein before generating and integrating the evaluationvalues for each site of the main subject together, the image recognitionsection determines the evaluation values for a obstructing areacorresponding to the subject of interest to indicate a part of the mainsubject.
 7. The focusing device according to claim 1, furthercomprising: an image generating section configured to, after thefocusing estimation section carries out the focusing estimation,separate one of the plurality of image data into two image data based ona position of the subject of interest and carry out image processing onthe two image data to acquire image data free from the subject ofinterest.
 8. The focusing device according to claim 2, furthercomprising: an image generating section configured to, after thefocusing estimation section carries out the focusing estimation,separate one of the plurality of image data into two image data based ona position of the subject of interest and carry out image processing onthe two image data to acquire image data free from the subject ofinterest.
 9. The focusing device according to claim 3, furthercomprising: an image generating section configured to, after thefocusing estimation section carries out the focusing estimation,separate one of the plurality of image data into two image data based ona position of the subject of interest and carry out image processing onthe two image data to acquire image data free from the subject ofinterest.
 10. The focusing device according to claim 4, furthercomprising: an image generating section configured to, after thefocusing estimation section carries out the focusing estimation,separate one of the plurality of image data into two image data based ona position of the subject of interest and carry out image processing onthe two image data to acquire image data free from the subject ofinterest.
 11. An image pickup apparatus comprising: a distanceinformation acquisition section configured to acquire information on adistance to a subject based on a plurality of image data acquired bytaking images of the subject at a plurality of different viewpointpositions; a subject detection section configured to detect a first areaand a second area based on the distance information acquired by thedistance information acquisition section and threshold information forthe distance range within which a subject of interest that obstructs amain subject in the subject is present, the subject of interest beingpresent in the first area in each of the plurality of image data, thesubject of interest not being present in the second area; a focusingestimation section configured to carry out the focusing estimation onthe main subject based on the distance information on the second area ineach of the plurality of image data; and an image generating sectionconfigured to acquire image data by taking images of the subjectincluding the main subject on which the focusing estimation has beencarried out by the focusing estimation section.
 12. The image pickupapparatus according to claim 11, wherein the distance informationacquisition section sequentially acquires the distance information foreach of the plurality of image data acquired by consecutively takingimages of the subject at the viewpoint positions, the subject detectionsection detects the first area and the second area in each of theplurality of consecutively acquired image data, and the focusingestimation section sequentially carries out the focusing estimation onthe main subject based on the distance information on the second areasequentially detected by the subject detection section.
 13. The imagepickup apparatus according to claim 11, wherein the focusing estimationsection carries out the focusing estimation on the main subject by hillclimbing using the image data on the second area.
 14. The image pickupapparatus according to claim 11, further comprising: an imagerecognition section configured to generate a plurality of evaluationvalues using, as an index, similarity of each of a plurality of featurepoints of the main subject in the plurality of image data, integrate theevaluation values together, and recognize whether or not the second areacorresponds to the main subject based on the evaluation values for thesecond area after the integration process.
 15. The image pickupapparatus according to claim 14, wherein the image recognition sectiongenerates and integrates the evaluation values for each site of the mainsubject together.
 16. The image pickup apparatus according to claim 14,wherein before generating and integrating the evaluation values for eachsite of the main subject together, the image recognition sectiondetermines the evaluation values for a obstructing area corresponding tothe subject of interest to indicate a part of the main subject.
 17. Theimage pickup apparatus according to claim 11, further comprising: animage generating section configured to, after the focusing estimationsection carries out the focusing estimation, separate one of theplurality of image data into two image data based on a position of thesubject of interest and carry out image processing on the two image datato acquire image data free from the subject of interest.
 18. The imagepickup apparatus according to claim 12, further comprising: an imagegenerating section configured to, after the focusing estimation sectioncarries out the focusing estimation, separate one of the plurality ofimage data into two image data based on a position of the subject ofinterest and carry out image processing on the two image data to acquireimage data free from the subject of interest.
 19. The image pickupapparatus according to claim 13, further comprising: an image generatingsection configured to, after the focusing estimation section carries outthe focusing estimation, separate one of the plurality of image datainto two image data based on a position of the subject of interest andcarry out image processing on the two image data to acquire image datafree from the subject of interest.
 20. The image pickup apparatusaccording to claim 14, further comprising: an image generating sectionconfigured to, after the focusing estimation section carries out thefocusing estimation, separate one of the plurality of image data intotwo image data based on a position of the subject of interest and carryout image processing on the two image data to acquire image data freefrom the subject of interest.